Codecov Report

Merging #854 (a8be8a2) into main (6ccb154) will increase coverage by 0.00%.
The diff coverage is 75.00%.

@@           Coverage Diff           @@
##             main     #854   +/-   ##
=======================================
  Coverage   71.49%   71.50%           
=======================================
  Files         335      335           
  Lines       17910    17906    -4     
=======================================
- Hits        12805    12803    -2     
+ Misses       5105     5103    -2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ccb154...a8be8a2. Read the comment docs.

Thanks for looking into this. I don't quite understand how this improves the API though. I guess we no longer have to import the schema using the FFI? But how can we pass the data type between arrow implementations in a portable way? Is it just an int enum and therefore this isn't an issue?

Also I don't understand how you can pass the data type into the C function that expects a schema pointer. How does that work at all? Are we not losing information stored in the Schema as well?

My comment was mostly about wanting a RecordBatch struct that bundled the Schema with the Arrays. Is the idea with this PR that the Schema isn't needed, and so a Chunk is therefore equivalent to a RecordBatch?

Is the idea with this PR that the Schema isn't needed, and so a Chunk is therefore equivalent to a RecordBatch?

Yes, that is the core observation - exporting an Array does not require a schema nor fields, only its datatype. Because every array stores such a datatype in this implementation of Arrow, we can just expose the API for that.

This does not close #854 , they are related but independent :)

Also I don't understand how you can pass the data type into the C function that expects a schema pointer. How does that work at all? Are we not losing information stored in the Schema as well?

What is "the C function"? What is "Schema"?

There are two things supported by the C data interface:

1. A Field (Ffi_ArrowSchema)
2. An Array (Ffi_ArrowArray)